1. Field of the Invention
The present invention relates to a camera calibration method and device for calculating parameters representative of the characteristics of a camera and, more particularly, to such a method and device adapted for calculating parameters relative to a camera of a type which shoots an object to pick up an image thereof and outputs electronic image data. In addition, the present invention relates to a camera calibration method and device for calculating parameters necessary for distance measurement in stereographic camera calibration by the use of an arbitrary plane not restricted with regard to any spatial position.
2. Description of the Prior Art
Most of the existing cameras are so designed as to perform center projection with a pinhole camera model. Center projection is capable of forming a projected image by disposing the chroma value of a point M on the surface of a three-dimensional object, at an intersection of a straight line (termed “sight line” also), which passes through a projection center C and the point M on the object surface, and a projection screen of the camera. In such center projection, the image of an object is projected to be larger with approach to the projection center C of the camera, and the image is projected to be smaller with receding from the projection center C while the object is dimensionally the same.
When the object to be shot is a plane, it is obvious, geometric-optically, that the image obtained by shooting the object in an oblique direction to the front thereof becomes a projected image formed through projective transformation of the image shot from a position opposite to the front. In the technical field of image processing, it is widely known that a projected image is obtained through projective transformation of a front image by a projective transformation matrix H. For example, in case a front image is composed of electronic image data captured by a digital camera, a projected image equal to one shot from a desired direction (sight line) can be calculated relatively fast with facility through projective transformation of the captured front image. For example, in Kenichi Kanetani, “Image Understanding” (Morikita Shuppan, 1990), there is described a point where an original image can be transformed by a projective transformation matrix into an image visible at a different angle. Such projective transformation of a shot image can be performed fast by the use of, e.g., computer resources.
The geometric-optic attribute related to projective transformation also can be applied to distance measurement of an object based on “stereography.” Here, the stereography is defined as a method of measuring, according to the principle of trigonometrical measurement, the distance between the projection center and each point of a scene (i.e., an image to be shot by the use of images shot) from a number of view points (projection centers) having a predetermined positional relationship.
It is supposed in this specification that, for the sake of explanatory convenience, stereography is carried out by the use of two cameras representing two view points. One camera is employed as a base camera which shoots an object from a position exactly opposite to the front thereof and then outputs a base image. Meanwhile, the other camera is employed as a detection camera which shoots the object from an oblique direction and then outputs a detection image. FIG. 7 typically shows how the base camera and the detection camera are disposed with respect to the image plane, and FIGS. 8A and 8B typically show a base image and a detection image obtained by shooting a substantially square pattern with the base camera and the detection camera, respectively.
In the image picked up by the base camera, as shown in FIG. 7, a point M is observed at an intersection m of a straight line, which passes through a point M on the object plane to be shot and a projection center Cb of the base camera, with a projection screen Sb of the base camera. The straight line passing through the point M and the projection center Cb denotes a sight line of the base camera. Meanwhile, in the image picked up by the detection camera, a point M is observed at an intersection m′ of a straight line, which passes through the point M and a projection center Cb of the detection camera, with a projection screen Sd of the detection camera. The straight line passing through the point M and the projection center Cd of the detection camera denotes a sight line of the detection camera.
The sight line of the base camera is observed as a straight line on the projection screen of the detection camera, and this straight line is termed “epipolar line.”
In the example shown in FIGS. 7, 8A and 8B, the image shot by the base camera positioned exactly opposite the substantially square pattern becomes a square, whereas the image shot by the detection camera positioned obliquely to the pattern appears to be trapezoidal since the side having a longer distance from the view point is contracted. This is based on the fundamental characteristic of center projection that any object of the same dimensions is projected to form a larger image with approach to the projection center C of the camera, and is projected to be smaller with receding from the projection center C.
As described above, in case the object to be shot is a plane, the image from the detection camera corresponds to an image obtained through projective transformation of the image from the base camera. That is, the following condition is satisfied between the point m (xb, yb) in the image from the base camera and the point m′ (Xd, yd) in the image from the detection camera. In the equation given below, H denotes a 3×3 projective transformation matrix.m′=H′m
The projective transformation matrix H implicitly includes internal and external parameters of the camera and a plane equation, and has eight degrees of freedom which are left in scale factors. In “Image Understanding” written by Kenichi Kanetani (Morikita Shuppan, 1990), there is a description that, between a base image and a reference image to be compared, points of mutual correspondence can be found through projective transformation.
The sight line of the base camera appears as a straight line termed an epipolar line on the projection screen of the detection camera (as explained above with reference to FIG. 7). The point M existing on the sight line of the base camera appears at the same observation point m on the projection screen of the base camera, regardless of the depth of point M; i.e., the distance to the base camera. Meanwhile, the observation point m′ of point M on the projection screen of the detection camera appears, on the epipolar line, at a position proportional to the distance between the base camera and the observation point M.
FIG. 9 illustrates the epipolar line and the observation points m′ on the projection screen of the detection camera. As shown in this diagram, the observation point in the reference image to be compared is shifted to m′1, m′2, m′3 in accordance with positional changes of the point M to M1, M2, M3. In other words, the position on the epipolar line corresponds to the depth of the observation point M.
The observation point m′ relevant to the observation point m of the base camera is searched on the epipolar line by utilizing the geometric-optic attribute mentioned above, so that it becomes possible to measure the distance between the base camera and the point P. This is the fundamental principle of “stereography.”
However, generation of a perspective image on the basis of the front image obtained by actually shooting a desired object, or measurement of the distance to the object from a number of images obtained by a number of cameras according to stereography, is premised on the condition that the imaging optical system of each camera has a characteristic which is completely coincident with a theoretical one. For this reason, it is necessary to execute predetermined correction with regard to the image acquired by actual shooting. For example, a camera lens generally has a distortion parameter, and its observation point in the image is positionally displaced from a theoretical one. Therefore, unless calculating the parameter peculiar to the camera and correcting the image data in accordance with such parameter in projective transformation, it is impossible to obtain an accurate projected image from the front image and, consequently, accurate measurement of the depth fails to be carried out by stereography.
The camera has, in addition to such a distortion parameter of its lens, internal parameters representing the camera characteristics, and external parameters representing the three-dimensional position of the camera. Generally, a method of calculating the camera parameters is termed “camera calibration.” Although a variety of techniques for camera calibration have been proposed heretofore, there is no established one in the present circumstances. In general, some exclusive appliances and so forth are required due to restriction of data for calibration, thus rendering the processing very complicated.
The most typical camera calibration is a method of first shooting a calibration pattern which consists of a number of reference points whose three-dimensional spatial positions are known, and then calculating the entire camera parameters simultaneously inclusive of internal parameters, external parameters and distortion parameter. The technique of such camera calibration is disclosed in, e.g., Roger Y. Tsai, “An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision” (1986, IEEE). However, for carrying out the method proposed by Tsai, it is necessary to prepare a calibration pattern where accurate reference points are plotted. Further, a mechanism for exactly positioning the reference points is also required.
In order to reduce such restrictions of data, there are recently proposed some improved camera calibration methods suing an arbitrary plane without any positional restriction.
For example, a method of calculating internal parameters of a camera is disclosed in Zhengyou Zhang, “A Flexible New Technique for Camera Calibration”, Microsoft Research Technical Report, 1999 (http://www.research.microsoft.com/zhang/), wherein no description is given for providing parameters relative to stereography.
Briefing the above, it is highly demanded now to create an improved procedure which carries out camera calibration in a stereo system merely by shooting a known planar pattern with cameras from different directions without any positional restriction on the plane.